Validation of Psychometric Instrumentswith Classical Test Theory in Social and Health Sciences: A practical guide / Validación de instrumentos psicométricos en ciencias sociales y de la salud: una guía práctica

Recientemente se ha incrementado significativamente el número de estudios psicométricos junto a avances estadísticos cruciales para evaluar las evidencias defiabilidad y validez de los testsy escalas de medida. Dada la importancia de proporcionar procedimientos más exactos tanto en la metodología como en la interpretación de las puntuaciones, los editores de la revista Anales de Psicología proponen esta guía para abordar los tópicos más relevantes en el campo de la psicometría aplicada. Con esta finalidad, el presente manuscrito analiza los tópicos principales de la TeoríaClásica de Tests (e.g., análisis factorial exploratorio/confirmatorio, fiabilidad, validez, sesgo, etc.) con vistas a sintetizar y clarificarlas aplicaciones prácticas, y mejorar los estándares de publicación de estos trabajos.(AU)

In recent years,there has been a significant rise in the number of psychometric studies, together with crucial statistical advances for validity and reliability measures. Given the importance of providingaccurate procedures both in methodology and score interpretationof tests and/or measurement scales, the editors-in-chie fof the journal Annals of Psychology have draftedthis guide to address the most relevant issues in the field of applied psychometry. To this end, the present manuscript analyses the main topics under the Classical Test Theory framework (e.g., exploratory/confirmatory factor analysis; reliability, bias, etc.) aimingto synthesizeand clarifythe best practical applications;and improve publication standards.(AU)

A multi-group confirmatory factor analysis of the revised children's anxiety and depression scale (RCADS) in Spain, Chile and Sweden.

BACKGROUND: There is a need for a measure that can be used across countries and cultures to advance cross-cultural research about internalizing mental health symptoms in children and adolescents. The Revised Children's Anxiety and Depression Scale (RCADS) is a potential candidate, but no study has examined whether its scales are measured similarly in youth populations from different countries. METHODS: In this study, we use confirmatory factor analysis (CFA) and multi-group CFA to examine the cross-cultural properties of a short and free to use 30-item version of RCADS that assesses social, generalized, panic, and separation anxiety alongside depression and obsessive-compulsive symptoms. We tested the factor structure of RCADS in children and adolescents from Chile, Spain, and Sweden, recruited using different research designs (i.e., school-based studies and an anonymous web survey), and whether the factor structure showed measurement invariance across the three countries. RESULTS: The proposed factor structure of RCADS showed good model/data fit in all three countries and was superior to a unidimensional model in which correlations among scale items were explained by a single broad internalizing factor. Each RCADS subscale showed adequate to excellent internal consistency in all three countries and multi-group CFA supported scalar invariance across the three countries. LIMITATIONS: No clinical sample was included. CONCLUSIONS: This study provides an important first step in supporting the use of RCADS in cross-cultural research on depression, anxiety and obsessive-compulsive symptoms in children and adolescents, but more work on validity aspects of the scale across cultures is needed.

A Meta-analysis of Gut Microbiota in Children with Autism.

Previous studies have reported dysbiosis in the gut microbiota (GM) of children with autism spectrum disorders (ASD), which may be a determining factor on child development through the microbiota-gut-brain axis. However, it is not clear if there is a specific group of dysbiotic bacteria in ASD. The aim of this study was to carry out a meta-analysis on the studies that analyze GM in children with ASD. 18 studies fulfilled our selection criteria. Our results showed a lower relative abundance of Streptococcus (SMD+ = - 0.999; 95% CI - 1.549, - 0.449) and Bifidobacterium genera (SMD+ = - 0.513; 95% CI - 0.953, - 0.073) in children with ASD. Overall, the Bifidobacterium genera is involved. However, differences found between studies are attributed to factors such as reporting bias.

Identification of emotions and physiological response in individuals with moderate intellectual disability.

According to Ekman's model, the basic emotions are anger, disgust, fear, happiness, sadness, and surprise. These emotions are universal and have an adaptative function. However, studies on these emotions among individuals with moderate intellectual disability (MID) are limited, mainly owing to issues in sample size and sample homogeneity. The present study aimed to explore differences between a MID group (n = 10) and a non-MID control group (n = 10) in 1) the identification of basic emotions; and 2) the physiological response with images related to fear, happiness, sadness, anger, and disgust, as well as images with neutral and sexual content. A total of 29 images from The International Affective Picture System and a biofeedback measurement system were used. The results showed no statistically significant differences in the identification of the basic emotions of fear, sadness, and disgust. However, statistically significant differences were found for happiness (p = .008), anger (p = .02), and neutral images (p = .02), and minor statistically significant differences were found for images with sexual content (p < .01) in MID individuals with respect to the control group. The MID group's physiological response was lower compared with the control group, with statistically significant differences for all emotional visual stimuli. This preliminary study showed important and interesting results for the study of emotions and sexuality in MID. Future studies should consider the sample size, sex and age comparisons, and the different levels of intellectual disability.

How Mentors Think About the Attainability of Mentoring Goals: The Impact of Mentoring Type and Mentoring Context on the Anticipated Regulatory Network and Regulatory Resources of Potential Mentors for School Mentoring Programs.

Research shows that trained mentors achieve better results than untrained ones. Their training should particularly address their expectations for their future mentoring. Our study involved 190 preservice teachers, potential mentors of ongoing school mentoring for primary and secondary school students of all grades. They were randomly assigned to one of four conditions in a 2-x-2 between-subjects design of mentoring type (traditional mentoring versus e-mentoring) and mentoring context (non-pandemic versus COVID-19 pandemic). Participants assessed mentoring conducted under these four conditions in terms of its appropriateness for achieving four mentoring program targets: learning, key skills, social targets, and problem coping. Participants were also asked to assess the resources available to achieve each program target. Overall, the potential mentors considered the various conditions to be suitable for achieving the four program targets. They were particularly favorable in their assessment of the possibility for the realization of learning targets. Likewise, they assumed that sufficient resources were available to achieve the targets. However, a repeated-measures MANOVA showed that the potential mentors considered more ambitious targets to be possible in traditional mentoring than in e-mentoring and normal (i.e., pre-pandemic) contextual conditions than during the COVID-19 pandemic. In contrast, they estimated the resources available to achieve the targets to be about the same in the four conditions. This indicates a decoupling of mentoring targets from the consideration of the resources needed to achieve them. This assumption was confirmed in correlation analyses and has implications for mentor training.

Psychometric Properties of the Spanish Version of the Goal Orientation Scales in Ecuadorian Undergraduate Students.

The present study aims to analyze the psychometric properties of the Goal orientation Scales (GOS; Skaalvik, 1997) in a sample of 2,170 Ecuadorian undergraduate students (M = 21. 97, SD = 3.61; 61.6% female). The Exploratory Factor Analysis and Confirmatory Factor Analysis supported the four-factor structure of the GOS, and the scale exhibited an adequate factorial invariance for gender. The multidimensional Rasch analysis revealed that one item showed misfit, and the distribution of items did not correspond well with the levels of achievement goals. The current research addresses a formal gap related to the validation of the GOS in a Latin American country and provides advanced psychometric information to further improve the scale for its application to Spanish-speaking samples.

University Entrance Examinations in Spain: Using the Construct Comparability Approach to Analyze Standards Quality.

In recent years, important methodological attempts have been made to explore the comparability of examination standards, especially in the context of certifications and university entrance. The present study aimed to explore the use of a construct comparability approach through a comparative analysis of the academic scores on 15 subjects from Spanish University Entrance Examinations in the Valencian Community, with a sample of 22,996 students in the call of June 2018. We employed the Rasch partial credit model as an estimation method, counting each subject as the item of an instrument related to academic achievement. The results confirmed the unidimensionality assumption and the goodness of fit of the model in relation to all subjects, although no discrimination between high and low ability students was detected because of the lack of monotonicity of the score categories. We observed that the level of difficulty of the subjects was appropriate to the students' ability levels. Important conclusions have been drawn for the improvement of the standard qualification process, and future research directions have been proposed.

Differences in Personal, Familial, Social, and School Factors Between Underachieving and Non-underachieving Gifted Secondary Students.

Using various identification methods, differences between underachieving and non-underachieving gifted students in personal, familial, social, and school variables were analyzed in a sample of 164 gifted students with IQs of 120 or higher; the sample was drawn from a larger sample of 1,400 compulsory secondary education students. Three procedures for identifying underachieving students were used: the standardized difference method, the regression method, and the Rasch method. The different profiles of underachieving and non-underachieving students in the personal, familial, social, and school variables were compared using MANOVA and ANOVA tests. Results revealed that underachieving gifted students scored significantly lower in learning strategies, goal orientations, self-concept, attitudes toward teachers, and perceived parent involvement in school variables. These results have clear educational implications as a result of identifying differences in non-cognitive factors.

Relationship between parent involvement and academic achievement through metacognitive strategies: A multiple multilevel mediation analysis.

BACKGROUND: Different studies reveal an association between parent involvement, metacognition, and academic achievement; however, the majority analyse the developmental process of acquisition in experimental or quasi-experimental designs with students enrolled in child education. Adolescence is an important stage in personal and academic development. Given the complexity of learning, and according to the metacognitive and affective model of self-regulated learning (Efklides, 2011, Educational Psychologist, 46, 6), different types of metacognitive processes are relevant for the optimal development of individuals in academic contexts. At the same time, individuals need stimuli from the environment, based on observation of their own and other's behaviour, as well as through communication and interaction with others. AIM: The present study examined the relations among parent involvement, metacognitive strategies, and academic achievement to investigate the mediational role of metacognition in the relation between parent involvement and academic achievement. SAMPLE: The participants were 1,398 high school students from Spain (47% female, M = 12. 5 years). METHOD: Multiple multilevel mediation analyses with Monte Carlo confidence intervals were used for measuring within-subjects effects at the student level and between-subjects effects at the class level. Parent involvement (perception of support, organization, and interest in the educational process; expectations; school relationship; time of support with homework) and metacognition were measured by questionnaires, whereas academic achievement was assessed using the end-of-term grades obtained by students for nine subjects. RESULTS: First, there was a significant direct effect of most parent involvement constructs on the mediational and dependent variables at both the within and between levels. Expectations had the highest predictive power on academic achievement. On the other hand, metacognitive strategies were an important mediator for all parent involvement constructs at both levels of analysis. CONCLUSIONS: These results highlight the importance of metacognition during early adolescence and suggest that parent involvement is crucial for the future development of educational models.

Validation of the Spanish adaptation of the School Attitude Assessment Survey-Revised using multidimensional Rasch analysis / Validación de la adaptación española del School Attitude Assessment Survey-Revised mediante el modelo de Rasch multidimensional

The School Attitude Assessment Survey-Revised (SAAS-R) was developed by McCoach and Siegle (2003b) and validated in Spain by Miñano, Castejón, and Gilar (2014) using Classical Test Theory. The objective of the current research is to validate SAAS-R using multidimensional Rasch analysis. Data were collected from 1398 students attending different high schools. Principal Component Analysis supported the multidimensional SAAS-R. The item difficulty and person ability were calibrated along the same latent trait scale. 10 items were removed from the scale due to misfit with the Rasch model. Differential Item Functioning revealed no significant differences across gender for the remaining 25 items. The 7- category rating scale structure did not function well, and the subscale goal valuation obtained low reliability values. The multidimensional Rasch model supported 25 item-scale SAAS-R measures from five latent factors. Therefore, the advantages of multidimensional Rasch analysis are demonstrated in this study

Validación de la adaptación española del School Attitude Assessment Survey-Revised mediante el modelo de Rasch multidimensional. Resumen: El School Attitude Assessment Survey-Revised (SAAS-R) fue desarrollado por McCoach y Siegle (2003b) y validado en España por Mi- ñano, Castejón, y Gilar (2014) a través del Modelo Clásico de Test. El objetivo del presente estudio es validar el SAAS-R a partir del análisis de Rasch Multidimensional. Los datos se obtuvieron de 1398 estudiantes que asistían a diferentes institutos de Educación Secundaria. El Análisis de Componentes Principales apoyó el modelo Rasch multidimensional. Se calibraron los parámetros de dificultad de los ítems y habilidad de los sujetos a partir de la misma escala latente. Se eliminaron 10 ítems por mostrar desajuste al modelo de Rasch. El Funcionamiento Diferencial del Ítem no mostró diferencias significativas de género con los 25 ítems restantes. La estructura escalar de 7 categorías no mostró un funcionamiento óptimo, y la subescala Valoración de Logro obtuvo niveles bajos de fiabilidad. El modelo Rasch multidimensional apoyó la escala SAAS-R con 25 ítems y 5 factores latentes. De esta forma, se demuestran las ventajas del modelo de Rasch multidimensional en el presente estudio

Differences in Learning Strategies, Goal Orientations, and Self-Concept between Overachieving, Normal-Achieving, and Underachieving Secondary Students.

The aims of this work were to identify and establish differential characteristics in learning strategies, goal orientations, and self-concept between overachieving, normal-achieving and underachieving secondary students. A total of 1400 Spanish first and second year high school students from the South-East geographical area participated in this study. Three groups of students were established: a group with underachieving students, a group with a normal level of achievement, and a third group with overachieving students. The students were assigned to each group depending on the residual punctuations obtained from a multiple regression analysis in which the punctuation of an IQ test was the predictor and a measure composed of the school grades of nine subjects was the criteria. The results of one-way ANOVA and the Games-Howell post-hoc test showed that underachieving students had significantly lower punctuations in all of the measures of learning strategies and learning goals, as well as all of the academic self-concept, personal self-concept, parental relationship, honesty, and personal stability factors. In contrast, overachieving students had higher punctuations than underachieving students in the same variables and higher punctuations than normal-achieving students in most of the variables in which significant differences were detected. These results have clear educational implications.

Estimation of the Proportion of Underachieving Students in Compulsory Secondary Education in Spain: An Application of the Rasch Model.

There are very few studies in Spain that treat underachievement rigorously, and those that do are typically related to gifted students. The present study examined the proportion of underachieving students using the Rasch measurement model. A sample of 643 first-year high school students (mean age = 12.09; SD = 0.47) from 8 schools in the province of Alicante (Spain) completed the Battery of Differential and General Skills (Badyg), and these students' General Points Average (GPAs) were recovered by teachers. Dichotomous and Partial credit Rasch models were performed. After adjusting the measurement instruments, the individual underachievement index provided a total sample of 181 underachieving students, or 28.14% of the total sample across the ability levels. This study confirms that the Rasch measurement model can accurately estimate the construct validity of both the intelligence test and the academic grades for the calculation of underachieving students. Furthermore, the present study constitutes a pioneer framework for the estimation of the prevalence of underachievement in Spain.

Academic Achievement in Early Adolescence: The Influence of Cognitive and Non-Cognitive Variables.

The present study examined the predictive effects of intellectual ability, self-concept, goal orientations, learning strategies, popularity and parent involvement on academic achievement. Hierarchical regression analysis and path analysis were performed among a sample of 1398 high school students (mean age = 12.5; SD =.67) from eight education centers from the province of Alicante (Spain). Cognitive and non-cognitive variables were measured using validated questionnaires, whereas academic achievement was assessed using end-of-term grades obtained by students in nine subjects. The results revealed significant predictive effects of all of the variables. The model proposed had a satisfactory fit, and all of the hypothesized relationships were significant. These findings support the importance of including non-cognitive variables along with cognitive variables when predicting a model of academic achievement.